Video Relationship Detection Using Mixture of Experts

Machine comprehension of visual information from images and videos by neural networks suffers two limitations: (1) the computational inference gap in vision language to accurately determine which object a given agent acts on then represent it language, (2) shortcoming stability generalization classifier trained single, monolithic network. To address these limitations, we propose MoE-VRD, novel approach relationship detection via mixture experts. MoE-VRD recognizes triplets form < subject,predicate,object > tuple extract between subject, predicate, processing. Since detecting subject (acting) object(s) (being acted upon) requires that action be recognized, base our network recent work detection. limitations associated with single networks, experts is based multiple small models, whose outputs are aggregated. That is, each expert learner capable tagging objects. employs an ensemble while preserving complexity cost original underlying model applying sparsely-gated experts, allows for conditional computation significant gain capacity. We show capabilities massive ability scale mixture-of-experts leads problem outperforms state-of-the-art.